SGI Performance Co-Pilot 2.1

home *** CD-ROM | disk | FTP | other *** search

/ SGI Performance Co-Pilot 2.1 / Performance Co-Pilot 2.1 for IRIX 5.3, 6.2, 6.3, 6.4 and 6.5.5.img / relnotes / pcp_eoe / ch3.z / ch3

Wrap

Text File | 1999-08-16 | 36KB | 776 lines

- 1 - 3. _C_h_a_n_g_e_s__a_n_d__A_d_d_i_t_i_o_n_s The major additions and changes for the basic services and tools of the Performance Co-Pilot are described in the following sections. Refer to the reference pages of the individual utilities for a complete description of any new functionality. 3.1 _I_n_f_r_a_s_t_r_u_c_t_u_r_e__C_h_a_n_g_e_s The following changes have been made to the PCP infrastructure that affect both collector and monitor configurations. 3.1.1 _P_C_P__2_._0__t_o__P_C_P__2_._1 1. To help with PCP deployments on systems running operating systems other than IRIX, the Performance Metrics Name Space (PMNS) has been overhauled to remove the iiiirrrriiiixxxx.... prefix from the names of the system-centric performance metrics, e.g. iiiirrrriiiixxxx....ddddiiiisssskkkk....ddddeeeevvvv....rrrreeeeaaaadddd____bbbbyyyytttteeeessss has become ddddiiiisssskkkk....ddddeeeevvvv....rrrreeeeaaaadddd____bbbbyyyytttteeeessss. In addition to changing the PMNS, translations are also handled dynamically in the PCP libraries, so all clients will continue to operate correctly using either the new or the old names. As a consequence no configuration files will need to be changed, and monitoring tools will work correctly in environments with a mixture of new and old style PMNS deployments. 2. The PCP inference engine ppppmmmmiiiieeee(1) has migrated from the _p_c_p._s_w._m_o_n_i_t_o_r subsystem to the _p_c_p__e_o_e._s_w._e_o_e subsystem, and the licensing restrictions have been relaxed to allow ppppmmmmiiiieeee to be used to monitor performance on the local host without any PCP licenses. 3. Support for running ppppmmmmiiiieeee(1) as a daemon has been added. This has many similarities to the ppppmmmmccccdddd(1) and ppppmmmmllllooooggggggggeeeerrrr(1) daemon support - ppppmmmmiiiieeee can be controlled through the cccchhhhkkkkccccoooonnnnffffiiiigggg(1) interface, and the startup and shutdown script, ////eeeettttcccc////iiiinnnniiiitttt....dddd////ppppmmmmiiiieeee which supports starting and stopping multiple ppppmmmmiiiieeee instances monitoring one or more hosts. This is achieved with the assistance of another script, ppppmmmmiiiieeee____cccchhhheeeecccckkkk(1) which is similar to the ppppmmmmllllooooggggggggeeeerrrr support script ppppmmmmllllooooggggggggeeeerrrr____cccchhhheeeecccckkkk(1). - 2 - 4. New capabilities have been added to assist in the estimation of PCP archive sizes. The ----rrrr option for ppppmmmmllllooooggggggggeeeerrrr(1) causes the size of the physical record(s) for each group of metrics and the expected contribution of the group to the size of the PCP archive for one full day of collection to be reported in the log file. The ----ssss option to ppppmmmmdddduuuummmmpppplllloooogggg(1) will report the size in bytes of each physical record in the archive. 5. Changes to ppppmmmmllllooooggggggggeeeerrrr(1) have greatly reduced the size of the *._m_e_t_a files created when logging metrics with instance domains that change over time. 6. As an aid to creating ppppmmmmllllooooggggggggeeeerrrr configuration files, ppppmmmmllllooooggggccccoooonnnnffff(1) is a new tool that allows selection of groups of commonly desired metrics and customization of ppppmmmmllllooooggggggggggggeeeerrrr configurations from a simple interactive dialog. 3.1.2 _P_C_P__1_._3__t_o__P_C_P__2_._0 1. The PCP Performance Metrics Name Space (PMNS) has previously been local to the application wishing to make reference to PCP metrics by name. In PCP 2.0 the PMNS operations are directed to the host or archive that is the source of the desired performance metrics, see ppppmmmmnnnnssss(4). The default name space is now associated with the PCP collector host rather than PCP monitor host. The distributed PMNS involves changes to both the PCP protocols between client applications and ppppmmmmccccdddd, and the internal format of PCP archive files (the ppppmmmmllllooooggggggggeeeerrrr(1) utility is able to write PCP archive logs in either the old or new formats). 2. PCP monitor hosts do not require any name space files, unless the local PCP client applications need to connect to PCP collector hosts that have not yet been upgraded to PCP 2.1. The distributed PMNS is implemented by having the name space functions of the PCP Performance Metrics Application Programming Interface (PMAPI) send and receive messages to and from the collector ppppmmmmccccdddd(1), just like the other PMAPI functions. 3. Full interoperability is supported for PCP 2.0, so new PCP components will operate correctly with either new or old PCP components. For example, a PCP client that connects to a PCP 1.x ppppmmmmccccdddd or attempts to process a - 3 - PCP archive created by a PCP 1.x ppppmmmmllllooooggggggggeeeerrrr will revert to using the local PMNS. The following table describes the supported interoperability between the client, ppppmmmmccccdddd(1) and PCP archive log components of PCP 2.0 and PCP 1.x: ___________________________________________________ PCP 1.x client PCP 2.0 client ______________________________________________________________________________________________________ PCP 1.x _p_m_c_d yes yes PCP 1.x archive yes yes PCP 2.0 _p_m_c_d yes yes PCP 2.0 archive no yes ___________________________________________________ |||||| |||||||||||| |||||| |||||| 4. The protocols between a PMDA and ppppmmmmccccdddd(1) have also changed, through a new version of the _l_i_b_p_c_p__p_m_d_a library. The following table describes the supported interoperability between the ppppmmmmccccdddd(1) and PMDA components of PCP 2.0 and PCP 1.x: ________________________________________________________________ PCP 1.x PCP 1.x PCP 2.0 PCP 2.0 daemon PMDA DSO PMDA daemon PMDA DSO PMDA ________________________________________________________________________________________________________________________________ PCP 1.x _p_m_c_d yes yes no no PCP 2.0 _p_m_c_d yes no yes yes ________________________________________________________________ ||||| |||||||||| ||||| ||||| ||||| ||||| 5. Product structural changes. The PCP product images have been re-arranged as follows. +o _p_c_p__e_o_e contains all of the pieces that are common to any successful PCP installation, particularly for a collector system, i.e. a place where ppppmmmmccccdddd is running. +o _p_c_p__e_o_e will ship in IRIX 6.5 and as part of PCP 2.0 for earlier IRIX releases. +o _p_c_p__e_o_e._s_w._m_o_n_i_t_o_r includes the oooovvvviiiieeeewwww monitoring application (for monitoring the performance of ORIGIN systems, from SC4-PCPORIGIN) and does not require any PCP licenses. +o _p_c_p._s_w._b_a_s_e provides the core PCP components that are outside _p_c_p__e_o_e. - 4 - +o _p_c_p._s_w._e_o_e for IRIX 6.2 or later has been replaced by _p_c_p__e_o_e._s_w._e_o_e (more formally the latter updates the former). For IRIX 5.3, _p_c_p._s_w._e_o_e is almost empty, but allows for clean upgrades from all earlier PCP versions. +o _p_c_p._s_w._m_o_n_i_t_o_r provides all of the PCP client tools for monitoring performance. +o The other _p_c_p subsystems provide optional PCP components. 6. PCP 2.0 introduces some new extensions to the PMAPI, mostly to accommodate the distributed PMNS and some underlying protocol changes to support enhanced inter-operability. The _l_i_b_p_c_p and _l_i_b_p_c_p__p_m_d_a libraries have been updated to version sgi2.0. The older versions of these libraries are still supported and may be installed from the _p_c_p._s_w._c_o_m_p_a_t subsystem if required for older PMDAs or customer-developed PCP applications. The API has also changed in some ways related to symbol hiding and changes in function arguments. The xxxxllllaaaatttteeee____ppppmmmmaaaappppiiii script in the _p_c_p__g_i_f_t_s._s_w._m_i_g_r_a_t_e subsystem may be used to automate the bulk of the source code translation from the 1.x version to the 2.0 version of the PMAPI, i.e. from _l_i_b_p_c_p._s_o._1 and _l_i_b_p_c_p__p_m_d_a._s_o._1 to _l_i_b_p_c_p._s_o._2 and _l_i_b_p_c_p__p_m_d_a._s_o._2. 7. In the transition from PCP 1.2 to PCP 1.3, the _l_i_b_p_c_p__l_i_t_e library was removed and the functionality replaced by the new PPPPMMMM____CCCCOOOONNNNTTTTEEEEXXXXTTTT____LLLLOOOOCCCCAAAALLLL option to ppppmmmmNNNNeeeewwwwCCCCoooonnnntttteeeexxxxtttt(3) in _l_i_b_p_c_p. The old _l_i_b_p_c_p__l_i_t_e library is still available in the _p_c_p._s_w._c_o_m_p_a_t subsystem. 8. ppppmmmmLLLLooooooookkkkuuuuppppNNNNaaaammmmeeee(3) now produces an extra error code, PPPPMMMM____EEEERRRRRRRR____NNNNOOOONNNNLLLLEEEEAAAAFFFF which means that the given name refers to a non-leaf node in the PMNS and so no PMID can be returned. Previously, if a non-leaf name was given then PPPPMMMM____EEEERRRRRRRR____NNNNAAAAMMMMEEEE would be returned. Now the error PPPPMMMM____EEEERRRRRRRR____NNNNAAAAMMMMEEEE means only that the name does not exist in the name space. 9. The function ppppmmmmGGGGeeeettttCCCChhhhiiiillllddddrrrreeeennnnSSSSttttaaaattttuuuussss(3) was added to the PMAPI. It was introduced mainly to satisfy a new need of ppppmmmmcccchhhhaaaarrrrtttt(1). As well as getting the names of all the children nodes, ppppmmmmGGGGeeeettttCCCChhhhiiiillllddddrrrreeeennnnSSSSttttaaaattttuuuussss returns a parallel array of the status of each node, indicating if it is a leaf or non-leaf node. - 5 - 10. A new (much smaller) format for PMDA help text files has been implemented, with support in the _l_i_b_p_c_p__p_m_d_a library and the nnnneeeewwwwhhhheeeellllpppp(1) utility. 11. A ----LLLL option was added to ppppmmmmdddduuuummmmpppplllloooogggg(1) to produce a more verbose variant of ----llll and report all of the details from a PCP archive label. 12. ppppmmmmllllooooggggggggeeeerrrr(1) uses the additional ppppmmmmccccdddd state change information to embed "mark" records in PCP archives when new PMDAs are started by ppppmmmmccccdddd. During replay, this prevents interpolation of values in the PCP archive across the life of an old and a new instance of the same PMDA. 13. The temporal index file (``.index'' suffix) is optional when PCP archives are being replayed. 14. Changes to track the object format of the booted IRIX kernel, briefly: +o ``MACH'' tag key PCP binaries in the images to ensure installation of o32, n32 or n64 versions as appropriate, based upon the installation platform and the IRIX version. +o Only one version of the ppppmmmmccccdddd binary will be installed, now in /_u_s_r/_e_t_c/_p_m_c_d. +o A new ppppmmmmoooobbbbjjjjssssttttyyyylllleeee(1) command is used to determine the appropriate kernel object style at run time as required, e.g. when ppppmmmmccccdddd(1) is attaching a DSO PMDA. 3.2 _C_o_l_l_e_c_t_o_r__C_h_a_n_g_e_s The following changes effect PMCD and the PMDAs that provide the collection services. 3.2.1 _P_C_P__2_._0__t_o__P_C_P__2_._1 1. The pppprrrroooocccc agent has been changed to use /_p_r_o_c/_p_i_n_f_o/_x_x_x_x if possible and only use /_p_r_o_c/_x_x_x_x if there is no alternative. Previously this agent always used /_p_r_o_c/_x_x_x_x to extract process information, and this caused unnecessary access checking to take place and some NFS contention problems were reported as a result. 2. The new eeeessssppppppppiiiinnnngggg PMDA provides quality of service metrics for consumption by the Embedded Support - 6 - Partner (ESP) infrastructure (released in IRIX 6.5.5). This PMDA can be used in conjunction with ppppmmmmiiiieeee(1) rules generated by the new ppppmmmmiiiieeeeccccoooonnnnffff(1) tool to detect service failure on either local or remote hosts. Among the services which can be probed are ICMP, SMTP, NNTP, ppppmmmmccccdddd, and local HIPPI interfaces using the new hhhhiiiipppppppprrrroooobbbbeeee(1) utility. 3. Changes to the way ppppmmmmccccdddd(1) and ppppmmmmllllooooggggggggeeeerrrr(1) are started from ////eeeettttcccc////iiiinnnniiiitttt....dddd////ppppccccpppp. a. When ppppmmmmllllooooggggggggeeeerrrr is chkconfig'd oooonnnn, ppppmmmmllllooooggggggggeeeerrrr instances are launched in the background from ////eeeettttcccc////iiiinnnniiiitttt....dddd////ppppccccpppp ssssttttaaaarrrrtttt, as this helps faster system reboots. In some cases this results in diagnostics from ppppmmmmllllooooggggggggeeeerrrr and/or ////uuuussssrrrr////ppppccccpppp////bbbbiiiinnnn////ppppmmmmllllooooggggggggeeeerrrr____cccchhhheeeecccckkkk that previously appeared when ////eeeettttcccc////iiiinnnniiiitttt....dddd////ppppccccpppp was run to now be generated asynchronously - any such messages are forwarded to the rrrrooooooootttt user as e-mail. These messages are in addition to those already written to /_v_a_r/_a_d_m/_p_c_p/_N_O_T_I_C_E_S by ppppmmmmppppoooosssstttt(1) from ////eeeettttcccc////iiiinnnniiiitttt....dddd////ppppccccpppp. b. A new utility, ppppmmmmccccdddd____wwwwaaaaiiiitttt(1), provides a more reliable mechanism for detecting that ppppmmmmccccdddd is ready to accept client connections. 4. In concert with changes to ppppmmmmiiiieeee, the ppppmmmmccccdddd PMDA has been extended to export information about executing ppppmmmmiiiieeee instances and their progress in terms of rule evaluations and action execution rates. Refer to the ppppmmmmccccdddd....ppppmmmmiiiieeee....**** metrics. 3.2.2 _P_C_P__1_._3__t_o__P_C_P__2_._0 1. ppppmmmmccccdddd(1) has been modified to support the distributed name space. This has meant the following changes: +o ppppmmmmccccdddd(1) now loads the default name space on startup or an alternative name space if specified by ----nnnn. +o On a SIGHUP signal it will reload the name space if it has been modified since the last load time. +o ppppmmmmccccdddd(1) responds to name space PDU requests using its local name space. 2. ppppmmmmccccdddd(1) no longer requires a PCP collector license to start, however connections will be refused for most - 7 - PCP clients unless the PCP collector license is installed and valid. Some clients (most notably oooovvvviiiieeeewwww(1)) can connect to ppppmmmmccccdddd independent of any PCP licenses. To support this, the client-pmcd connection protocols have been extended. 3. Support has been added for multiple protocol versions for the client-pmcd IPC and the interaction between ppppmmmmccccdddd and the PMDAs. 4. The logic used by ppppmmmmccccdddd for locating DSO PMDAs of the correct object format has been reworked. 5. The diagnostic event tracing options of ppppmmmmccccdddd have been extended to include an "unbuffered" mode where every event is reported as it happens, as opposed to buffering events in a ring buffer and only reporting on errors and in response to explicit requests. 6. The distributed PMNS changes mean that ppppmmmmccccdddd must load and export the PMNS in response to requests from client applications. ppppmmmmccccdddd's SIGHUP processing has been extended to include reloading the PMNS if the external PMNS file has been modified. 7. In concert with a change to ppppmmmmFFFFeeeettttcccchhhh in _l_i_b_p_c_p, ppppmmmmccccdddd is now able to export out-of-band information about ppppmmmmccccdddd state changes (specifically PMDA add, drop and restart) back to client applications. 3.3 _M_o_n_i_t_o_r__C_h_a_n_g_e_s The major additions and changes for the performance visualization and analysis tools are described below. 3.3.1 _P_C_P__2_._0__t_o__P_C_P__2_._1 1. ppppmmmmiiiieeee a. A syntactic restriction in the specification language has been relaxed, and actions may now have an arbitrary number of quoted arguments (previously at most two arguments were allowed). At the same time a problem with the ssssyyyysssslllloooogggg action was resolved, allowing the ----pppp option to be passed to llllooooggggggggeeeerrrr(1). For example, this is now valid: some_inst ( (100 * filesys.used / filesys.capacity) > 98 ) -> syslog "-p daemon.info 'file system close to full" " %h:[%i] %v% " "'"; - 8 - b. Metrics with dynamic instance domains are now correctly handled by ppppmmmmiiiieeee. Previously ppppmmmmiiiieeee instantiated the instance domain when it started, and was oblivious to any subsequent changes in the instance domain. This is most useful for rules using the metrics of the hhhhoooottttpppprrrroooocccc PMDA that is part of the ppppccccpppp product. c. The ppppmmmmiiiieeee language has been extended to allow two new operators mmmmaaaattttcccchhhh____iiiinnnnsssstttt and nnnnoooommmmaaaattttcccchhhh____iiiinnnnsssstttt that take a regular expression and a boolean expression. The result is the boolean AND of the expression and the result of matching (or not matching) the associated instance name against the regular expression. For example, this rule evaluates error rates on various 10BaseT Ethernet network interfaces (e.g. ecN, etN or efN): some_inst match_inst "^(ec|et|ef)" network.interface.total.errors > 10 count/sec -> syslog "Ethernet errors:" " %i"; The following rule evaluates available free space for all filesystems except the root filesystem: some_inst nomatch_inst "/dev/root" filesys.free < 10 Mbytes -> print "Low filesystem free (Mb):" " [%i]:%v"; d. During rule evaluation, ppppmmmmiiiieeee keeps track of the expected number of rule evaluations, number of rules actually evaluated, the number of predicates that are true and false, the number of actions executed, etc. These statistics are maintained as binary data structures in the mmmmmmmmaaaapppp'ed files /_v_a_r/_t_m_p/_p_m_i_e/<_p_i_d>. If ppppmmmmiiiieeee is running on a system with a PCP collector deployment, the ppppmmmmccccdddd PMDA exports these metrics via the new ppppmmmmccccdddd....ppppmmmmiiiieeee....**** group of metrics. e. Some restrictions on the expansion of macros (e.g. $name) have been removed, so macro expansion can occur anywhere in the ppppmmmmiiiieeee rule specifications. f. There has been some changes to improve the formatting of numeric values reported with the options ----vvvv, ----VVVV and ----WWWW, and for the expansion of %%%%vvvv in actions. In general terms these have - 9 - removed extra white space and reduced the likelihood of scientific notation being used. 2. A set of parameterized ppppmmmmiiiieeee rules have been developed which are applicable to most systems and will allow ppppmmmmiiiieeee to be used by new users without knowledge of the ppppmmmmiiiieeee syntax. A new utility, ppppmmmmiiiieeeeccccoooonnnnffff(1) has been built which allows these rules to be enabled or disabled, or the parameters and thresholds adjusted for a specific system. The combination of ppppmmmmiiiieeee, ppppmmmmiiiieeeeccccoooonnnnffff, ppppmmmmiiiieeee____cccchhhheeeecccckkkk and ////eeeettttcccc////iiiinnnniiiitttt....dddd////ppppmmmmiiiieeee provides the infrastructure required for PCP to search for behavior indicative of performance problems in a fully automated manner with little or no local customization required. Where customization is needed, ppppmmmmiiiieeeeccccoooonnnnffff(1) provides a convenient way of doing this. 3. A new utility, hhhhiiiipppppppprrrroooobbbbeeee(1), has been added which will check the status of HIPPI interfaces on a system. More sophisticated monitoring of HIPPI interfaces will be supported with a hhhhiiiippppppppiiii PMDA that will be released as part of the forthcoming PCP for HPC add-on product. 3.3.2 _P_C_P__1_._3__t_o__P_C_P__2_._0 1. Monitor applications no longer use the local default name space to get PMIDs for metric names unless the name space file is explicitly given as a ----nnnn option or the target ppppmmmmccccdddd, specified by ----hhhh, is a lower revision than PCP 2.1. Monitor applications use the distributed name space by sending PDU requests to ppppmmmmccccdddd. 2. To convert an existing user-written monitor application to use the distributed name space, the following must be done: +o Only call ppppmmmmLLLLooooaaaaddddNNNNaaaammmmeeeeSSSSppppaaaacccceeee(3) in the case of being given a ----nnnn option to specify a particular name space file. If one only wants to use the distributed name space then the call to ppppmmmmLLLLooooaaaaddddNNNNaaaammmmeeeeSSSSppppaaaacccceeee(3) can be deleted altogether. As soon as ppppmmmmLLLLooooaaaaddddNNNNaaaammmmeeeeSSSSppppaaaacccceeee(3) is called, the application is in local name space mode. +o All calls to the PMAPI name space functions must be made in a current PMAPI context. Previously, since a context was not used to service name space functions, it was possible to call these - 10 - functions before any new context was created. Now, if there is no current PMAPI context and ppppmmmmLLLLooooaaaaddddNNNNaaaammmmeeeeSSSSppppaaaacccceeee(3) has not previously been called when a name space function is called an error code will be returned. +o The calls to the PMAPI name space functions (in distributed mode) will take longer to execute (as they must send/receive a PDU to/from a remote ppppmmmmccccdddd). This means that the name space calls should not be made unnecessarily. +o All name space calls should be tested for failure. This was, of course, always the case, but previously there was more chance that one could get away without testing some calls for failure. However, now, every call could have some failure due to problems with sending or receiving a PDU. 3. The oooovvvviiiieeeewwww(1) application for monitoring Origin 200 and Origin 2000 systems has migrated from the SC4- PCPORIGIN product and is now included in the _p_c_p__e_o_e._s_w._m_o_n_i_t_o_r subsystem. This application is unlicensed and will operate with an unlicensed ppppmmmmccccdddd(1). 4. Many of the GUI PCP tools have been integrated into the IRIX Interactive Desktop environment, including launch by icon, drag-n-drop behavior for hosts or archives dropped onto PCP tools and a new ppppmmmmrrrruuuunnnn(1) command to support optional command line arguments for PCP tools launched from the desktop. 5. A PPPPeeeerrrrffffTTTToooooooollllssss page in the Icon Catalog has been created for launching PCP tools. 6. The handling of error messages for both GUI and command-line invocations of most tools has been unified and made more consistent. 7. The ppppmmmmttttiiiimmmmeeee(1) application that provides time control services to other PCP tools has been enhanced in a number of ways: +o A new Archive Time Bounds dialog (available only in archive mode) allows time window bounds to be constrained or expanded at run-time (particularly useful when replaying archives that are still growing). - 11 - +o Extensions to the ppppmmmmttttiiiimmmmeeee client protocol to allow clients to force VCR state changes and alter the time window bounds. +o The ppppmmmmttttiiiimmmmeeee protocol is nnnnooootttt backwards compatible with the PCP 1.x implementation, but all clients of ppppmmmmttttiiiimmmmeeee are re-released with the images on the PCP 2.0 CD and both ppppmmmmttttiiiimmmmeeee and its clients execute on the same system. 8. The ppppmmmmkkkkssssttttaaaatttt(1) command has been restructured in the wake of _l_i_b_p_c_p__l_i_t_e's demise, and in particular a new ----LLLL option supports stand alone use without ppppmmmmccccdddd(1). 3.4 _F_e_a_t_u_r_e_s__R_e_m_o_v_e_d__o_r__D_e_p_r_e_c_a_t_e_d In PCP 2.1, the following features from earlier PCP versions have been removed or are deprecated. 3.4.1 _P_C_P__1_._3__t_o__P_C_P__2_._0 1. The original intent in the Performance Co-Pilot framework was to support seamless VCR-replay between a current ``live'' source and a recently created archive source. In practice, the semantic and operational difficulties associated with transitions between ``live'' and ``archive'' modes are so significant that the feature has been abandoned. Earlier PCP releases included a ``stubbed-out'' application ppppmmmmvvvvccccrrrr(1) that did not really do anything and assorted infrastructure ``hooks''. These have been removed. 2. _l_i_b_p_c_p__l_i_t_e._s_o - replaced by PPPPMMMM____CCCCOOOONNNNTTTTEEEEXXXXTTTT____LLLLOOOOCCCCAAAALLLL The standalone (without ppppmmmmccccdddd) library _l_i_b_p_c_p__l_i_t_e._s_o has been replaced by the PPPPMMMM____CCCCOOOONNNNTTTTEEEEXXXXTTTT____LLLLOOOOCCCCAAAALLLL option to the ppppmmmmNNNNeeeewwwwCCCCoooonnnntttteeeexxxxtttt(3) routine.